Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix reimporting assets with csv in the project #92320

Merged

Conversation

Hilderin
Copy link
Contributor

@Hilderin Hilderin commented May 24, 2024

This should fix #83200

The problem was that the DebugDialogue.csv.import is invalid according to Godot because it’s missing the path line. Because of this the importer was not mark multithread and a counter was not incremented between threaded and non threaded reimportation. I fixed the increment problem in editor_file_system._reimport_file.

The problem with the invalid .import file was caused because ResourceImporterCSVTranslation is the only importer that has no extension. It does not write any file in the .godot folder. I fixed that adding a check in _get_path_and_type before returning ERR_FILE_CORRUPT.

@AThousandShips AThousandShips added the cherrypick:4.2 Considered for cherry-picking into a future 4.2.x release label May 24, 2024
@Hilderin Hilderin force-pushed the fix-importing-assets-with-csv branch from 4e18c87 to 9ad1d4d Compare May 24, 2024 15:42
@akien-mga akien-mga self-requested a review May 24, 2024 16:58
Comment on lines 2442 to 2450

// We need to increment the counter, maybe the next file is multithreaded
// and don't have the same importer.
from = i + 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part seems related to #85130, which I guess aims to address the same issue (albeit in another branch).

I couldn't get full confidence there that the change is correct. Would be good to re-review both PRs conjointly and figure out what this code is trying to do.

CC @RandomShaper @KoBeWi

Copy link
Contributor Author

@Hilderin Hilderin May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first tentative was exactly what was done in #85130 but the problem is that the variable 'from' is used elsewere after the Ref<ResourceImporter> importer = ResourceFormatImporter::get_singleton()->get_importer_by_name(reimport_files[from].importer);.

That is how I see it...

In my case I had:
Index 0: threaded = false, importer = csv_translation
Index 1: threaded = true, importer = texture
Index 2: threaded = true, importer = texture
Index 3: threaded = true, importer = texture
Index 4: threaded = false, importer = scene

If I use reimport_files[i].importer :
The first iteration, i = 0 and from = 0, _reimport_file was called directly
The second iteration, i = 1 and from = 0, it tries to start the thread because the importers of index 0 (from) and 2 (i+1) are different which is not was expected:
image

The thread will be started with the wrong index:
image

Edited: I looked more closely the comments on the #85130 from @RandomShaper and effectively, there is another potential problem that I did not saw when !importer.is_valid().
I'll look into it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referred to this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This have been addressed in this commit: 4f55118
I should have been clearer in my comment below.

@Hilderin Hilderin force-pushed the fix-importing-assets-with-csv branch from 9ad1d4d to 4f55118 Compare May 24, 2024 18:12
@Hilderin
Copy link
Contributor Author

This last commit included modifications to fix #85130 and the comments from @RandomShaper
I added another from = i + 1 if the reimport_file is in groups_to_reimport.
I don't known how to test it, I don't even known what are the groups for importation.

@Hilderin
Copy link
Contributor Author

I think this could fix at least partially the issue #58131
There were a real problem with the mulithreading when the project contains a csv file or invalid files. In the example of @ryanscottcausecast from #58131 the .import are linked to .stex file that Godot does not recognize. It's maybe linked to another issue with compressed textures?

With the fix, the preview are wrong because they are based on .import files and .godot folder but the texture are working fine:
image

@Hilderin Hilderin force-pushed the fix-importing-assets-with-csv branch from 4f55118 to 63ccccb Compare May 30, 2024 01:52
@Hilderin Hilderin force-pushed the fix-importing-assets-with-csv branch from 63ccccb to 760e20f Compare May 30, 2024 09:29
Copy link
Member

@AThousandShips AThousandShips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tested but the code looks good to me (can't speak for the last line of change that's discussed though)

@Hilderin
Copy link
Contributor Author

Sorry, @AThousandShips I'm not sure what you are referencing with:

can't speak for the last line of change that's discussed though

Everything should be good, I think.

@Hilderin Hilderin force-pushed the fix-importing-assets-with-csv branch from bce48a9 to f1099ab Compare June 11, 2024 21:04
@akien-mga akien-mga merged commit 74f9f12 into godotengine:master Jun 21, 2024
16 checks passed
@akien-mga
Copy link
Member

Welp I merged this inadvertently because I was testing it in my master branch before making a merge batch :D

But I guess it's a good way to confirm whether it's working or not, let's see how it fares from now to when I'd need to make beta3 builds.


Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug cherrypick:4.2 Considered for cherry-picking into a future 4.2.x release high priority topic:import
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(Re)importing Assets breaks apart on GLB/GLTF materials at clean project import because of an CSV file
4 participants